Algorithmic ("black box") trading systems consume market data that is used by software algorithms in the trading system to generate buy and sell requests to the market. The sequence of time stamped market data is referred to as a time series.
Historic market data is needed for testing the algorithms used by the trading system to make buy and sell decisions.
Time series databases can be used to store time series data and distribute the time series data to multiple trading applications. Specialized time series databases are used since relational databases generally show very poor performance for time series applications.
Time series database systems or streaming database systems intersect something called Complex Event Processing (frequently abbreviated as CEP). CEP involves running real-time algorithms on a data stream. Applications include finance and trading, along with intelligence applications.
David Luckham, who is a Professor Emeritus at Stanford runs the Complex Event Processing blog which includes discussion of industry and academic work on CEP and streaming databases.
Some time series databases are listed below:
Time series database software allows an application to fetch market time series data within a particular region. For example, a time series database will allow all IBM daily data to be fetched for every other trading day for 2007. Time series databases frequently support filter operations. In the data trades arrive at random intervals. Trading data can be consolidated ("chunked"), using an algorithm like average, for a particular interval. For example, the daily data for IBM could be returned in ten second intervals, averaging over the ten second region.
The flexibility of the time series databases comes at a significant financial cost. The software license fees for the time series databases I looked in the spring of 2008 at were all around $100,000. The market for these databases seems to be funds that do not do a lot of their own software development. Some databases, like Kx Systems provide interfaces for trading and financial modeling software.
One alternative to a time series database is to store daily time series data in a directory hierarcy. This is diagrammed below:
The time series files are stored in zip format. The Java open source TrueZIP package allows zip files to be opened like normal Java files and abstracts the details of file decompression.
Storing the time series files in compressed format reduces the amount of file I/O, which can greatly improve application performance.
Back to A Java Intra-day Traeding System, Quantitative Finance and other Topics